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| 7 | . Abstract 

■ This paper concerns singular value decomposition (SVD)-based computable formu- 
las and bounds for the condition number of the Total Least Squares (TLS) problem. 
For the TLS problem with the coefficient matrix A and the right-hand side b, a new 
closed formula is presented for the condition number. Unlike an important result in the 
literature that uses the SVDs of both A and [A, b], our formula only requires the SVD 
of [A, b] . Based on the closed formula, both lower and upper bounds for the condition 

■ number are derived. It is proved that they are always sharp and estimate the condition 
\^ , number accurately. A few lower and upper bounds are further established that involve 

at most the smallest two singular values of A and of [A, b]. Tightness of these bounds 
is discussed, and numerical experiments are presented to confirm our theory and to 
demonstrate the improvement of our upper bounds over the upper bound due to Golub 
and Van Loan. Such lower and upper bounds are particularly useful for large scale TLS 
problems since they require the computation of only a few singular values of A and of 
[A, b] other than all the singular values of them. 

Keywords: total least squares, perturbation, condition number, singular value decom- 
' position. 

AMS subject classification (2000): 65F35, 15A12, 15A18. 

1 Introduction 

For given A £ M mxn (m > n), b S M m , the TLS problem can be formulated as (see, e.g., 

mm\\[E, f]\\ F , subject to b + / G K{A + E), (1) 

where || • \\f denotes the Frobenius norm of a matrix and TZ(-) denotes the range space. 
Suppose that [Etls, Itls] solves the above problem. Then x = xtls that satisfies the 
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equation (A + Etls)x = b + frLS is called the TLS solution of (P). The TLS problem is a 
formulation of the linear approximation problem Ax ~ b. In this paper, we concentrate on 
the inconsistent linear approximation problem, i.e., b ^ 1Z(A). Otherwise, [Etls-, Itls] = 
O, the zero matrix. 

Given a problem, the condition number measures the worst-case sensitivity of its solution 
to small perturbations in the input data. It is well known that the condition number 
is independent of perturbations themselves and is expressed by some information on the 
original data. Combined with backward errors, it provides a (possibly approximate) linear 
upper bound for the forward error, i.e., the difference between a perturbed solution and the 
exact solution. Since the 1980's, algebraic perturbation analysis for the TLS problem has 
been made extensively; see [H [LH [20] and the references therein. From the expressions 
of perturbation bounds presented in [H [6j El [2Q], we can see that there are some essential 
distinctions between them and a standard form. The perturbation bound in [6j is a standard 
one in the sense that it is expressed as some perturbation free factor times backward errors. 
So, this factor is naturally an upper bound for the TLS condition number. The perturbation 
bound in [20j is nonstandard and unusual since the perturbation bound is not zero when a 
perturbation is exactly zero. Actually, a careful observation reveals that the bound is never 
less than a certain positive constant under the assumption b ^ 1Z(A). As a result, it makes 
no sense to extract an upper bound for the TLS condition number from this perturbation 
bound. The perturbations bounds in [HE] are very different from a standard perturbation 
bound in that they contain some information on the perturbed TLS problem, e.g., the 
TLS solution xtls of the perturbed TLS problem in [3] and the right singular vectors 
associated with the smallest singular values of both [A, b] and its perturbed matrix in |14j . 
Because of these features and the fact that the condition number itself has nothing to do 
with perturbations, it is impossible to extract upper bounds for the TLS condition number 
from the perturbation bounds in [HE]. If one attempts to find suitable upper bounds for 
the TLS condition number, some further and complicated treatments are required and it is 
necessary to make each of their perturbation bounds become a standard one, that is, some 
perturbation free factor times the backward error. 

In recent years, asymptotic perturbation analysis and TLS condition numbers have been 
investigated. The authors [13] and Zhou et al. [21] have presented a first order perturbation 
analysis of the TLS problem and established Kronecker products-based condition number 
formulas. Baboulin and Gratton [1] have derived a SVD-based closed formula for the TLS 
condition number and an upper bound for it that involve all the singular values and the 
right singular vectors of both A and [A, b] and several singular values of them, respectively. 
To our best knowledge, however, there has been no lower bound available for the TLS 
condition number in the literature. 

It is well known that the TLS solution xtls involves the smallest singular value and the 
corresponding right singular vector of [A, b], see, e.g., [6]. Based on this intimate relation 
and motivated by the work of pQ, we continue our work in [13] to study SVD-based TLS 
condition number theory. We will derive a number of results. Firstly, we establish a new 
closed formula of the TLS condition number. It is distinctive that, unlike the result in [T] 
that requires the SVDs of both A and [A, b], our formula only uses the singular values and 
the right singular vectors of [A, b]. Secondly, starting with the closed formula, we present 
both lower and upper bounds for the condition number that involve the singular values of 
[A, b] and the last entries of the right singular vectors of [A, b]. Furthermore, we prove that 
these bounds are always sharp and can estimate the condition number accurately. We then 
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focus on cheaply computable bounds for the TLS condition number. We establish lower 
and upper bounds that involve at most the smallest two singular values of A and of [A, b\. 
We discuss how tight the bounds are. These bounds are particularly useful for large scale 
TLS problems since they require to compute only a very few smallest singular values of A 
and [A, b] other than all the singular values of them. So we can compute these bounds by 
using some iterative solvers for large SVDs, e.g., |10|, 1 11| . Prom [6], as mentioned previously, 
an upper bound for the TLS condition number can be extracted. It has been simplified and 
applied to evaluate the conditioning of the TLS problem in [3]. We will present numerical 
experiments to demonstrate great improvements of our upper bounds over the upper bound 
due to Golub and Van Loan [6] . 

The paper is organized as follows. In Section 2, we present some preliminaries necessary. 
In Section 3, we establish some useful and necessary results related to a specific orthogonal 
matrix. In Section 4, we present a new closed formula of the TLS condition number. The 
straightforward bounds for the TLS condition number are derived in Section 5. In Section 6, 
we report numerical experiments to show the tightness of our bounds for the TLS condition 
number and considerable improvements over Golub- Van Loan's bound. We conclude the 
paper with some remarks and future work in Section 7. 

Throughout the paper, for given positive integers m, n, denote by M. n the space of n- 
dimensional real column vectors, by W nxn the space of all m x n real matrices, and by || • || 
and || • \\f 2-norm and Frobenius norm of their arguments, respectively. Given a matrix A, 
A(l : i, 1 : j) is a Matlab notation that denotes the submatrix in the intersection of rows 
1, . . . ,i and columns 1, . . . ,j, and cri(A) denotes the ith largest singular value of A. For 
a vector a, a(i) denotes the ith component of a, and diag(a) is a diagonal matrix whose 
diagonals are a(z)'s. I n denotes the n x n identity matrix and O m>n denotes the mx n zero 
matrix with O a zero matrix whose dimension is clear from the context. For the matrices 
A = [a\, . . . ,a n ] = [aij] G m mxn anc l B, A(g> B = [a^B] is the Kronecker product of A and 
B, and the linear operator vec : M. mxn — > M. mn is defined by vec(A) = [aj , . . . , a^] T . 

2 Preliminaries 

Throughout the paper, let A = C/diag(<Ti, . . . , 6- n )V T be the thin SVD of A £ R mxn , where 
<ri > ••• > o- n , U e R mxn , V G M nxn and U T U = I n , V T V = I n , and let [A, b] = 
[/diag(CTi,...,cr n+ i)T/ T be the thin SVD of [A, b] G R mx ("+ 1 ), where a x > ■ • • > a n+1 , 

U = [ Ul ,...,U n+1 ] G Rmx(n+l) ) y = . . . } £ R (n+l)x(n+l) and jjTjj = 

v T v = / n+1 . 

The TLS problem ([1]) may not have a solution, but it does have a unique solution if the 
following condition holds [T6] : 

A has rank n and b JL Mmin, (2) 

where U m i n denotes the left singular vector subspace of A corresponding to its smallest 
singular value. Throughout the paper, we always assume that (|2|) holds. 

It is noted in [16] that condition ([2]) means cr n +l < &n, the existence and uniqueness 
condition of the TLS solution given in [6]. Under the condition that a n +i < o n , it is proved 
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in [6] that 



XTLS 



(A T A - al +l I)-'A T b 

Un+l(l) v n+ i{n) 



v n+ i{n + 1) ' " ' ' v n+ i(n + 1) 



(3) 
(4) 



We comment that ([2|) implies that A T b / 0. So xtls 7^ 0. 

Given the TLS problem (HJ) , let A = A + AA, 6 = 6 + Ab, where A.4 and A6 denote the 
perturbations in A and b, respectively. Consider the perturbed TLS problem 



min||[£, f]\\ F subject to b + / £ + 



(5) 



Under the assumption that 6 ^ 7£(j4), it follows from ([2]) that < a n+ y < a n . In [13], the 
following result is established for the TLS solution xtls of the perturbed TLS problem ([5]). 

Theorem 1 Suppose that the TLS problem ([I]) satisfies < o~ n -yy < a n . Define r 



Axtls — b and G 



T 

v TLS-> 



-l]<g>im € R m x( mn + m ). //II [A A A6]|| F is small enough, then 



the perturbed problem ([5]) has a unique TLS solution xtls- Moreover, 



xtls = xtls + K 



vec(A^) 
Ab 



+ 0(\\[AA, Ab]\\ 2 F ) 



where 



K={A T A-a 2 n+1 I n 



-i 



2A T ^^—G-A T G- [/„ 



nnx(mn+m) 



It is shown in |6| that 



u. 



n+1 



1 + II^TLsll 2 



and 



From Q, it follows that 



A T r 



1 + \\XTLS 



V n +1 



\A + \\XTLS\\ 2 



J2 X TLS = 0- n+1 XTLS- 



XTLS 
-1 



(6) 

(7) 

(8) 
(9) 

(10) 



up to a sign ±1. We will use the above two relations later. The following basic properties 
of the Kronecker products of matrices can be found in [8] and are needed later: 



{Ay <g> A 3 )(A 2 <g> A 4 ) = (AyA 2 ) ® {A 3 A 4 ), 
(Ay ® A 2 ) T = A\ r ® A T , 

where Ai, i = 1, . . . , 4 are matrices with appropriate sizes. 
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3 Some results related to a specific orthogonal matrix 



In this section, we establish a number of results that are related to a specific orthogonal 
matrix. They play a central role in deriving our lower and upper bounds for the condition 
number of the TLS problem in the next section. 

Proposition 1 Let W be an arbitrary (n + 1) x (n + 1) orthogonal matrix with W(n + 
l 3 n + 1) = -a, < a < 1. Denote W\\ = W(l : n, 1 : n). Then 



ffi (Wn) = • • • = o n -i{Wu) = 1, a n {Wn) = a. 
Furthermore, W can be written as 



w 



\/T— a* v 



2 t,t 

n 



yl — a 2 u r 
—a 



:n) 



(12) 



where u n and v n are the left and right singular vectors associated with the smallest singular 
value of Wxx ■ 



Proof. It is an immediate result of Theorem 2.6.3 in [7]. 

Let \J3i, . . . , P n , — a] be the last row of W. From (fT2j) we have 

1 



□ 



a' 



(13) 



W 



Since (a x , n n , v n ) is the largest singular triplet of W n T ', from (fTT]) we get the SVD of 

-T. 
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n-1 



iy n T = a l u n vl + ^2' 



1=1 



where Ui,Vi,i = 1, ... ,n— 1 represent the left and right singular vectors associated with 
the singular value one. Then, by (|13p we obtain 



W 



ii 



[a u n ,ui,...,u n -i] 



^i(l) 



vAn) 



U n + Wl,. 



■Wn-i(l) ••• v n -i(n) 



' y/l — at- 



-.u n + w n 



where Vi(k) denotes the kth component of V{ and 



n— 1 

Wfc = X \vj{k)uj, k = l,...,n. 
i=i 



From (1141) we get 



which will be used later. 

Before proceeding, we need the following lemma. 



n-l 



(14) 



(15) 



(16) 



(17) 
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Lemma 1 For given matrices A U A 2 G R nxn , if A\A 2 = O, then 
k\\Ai\\ + \\A 2 \\) < \\A 1 +A 2 \\ < PJ + \\A 2 \\. 



(18) 



Proof. The upper bound in (|18p is obvious. It suffices to prove the lower one. For an 
arbitrary vector x G 1", from (A\x) T (A 2 x) = it follows that 



||Aix||, p 2 x|| < \\A lX + A 2 x\\ 



and that 



Pi|| = max|| x || = iPix|| < max|| x || =1 pix + A 2 x\\ = pi + A 2 \\, 
p 2 || = maxi^i^ipaxll < max|| x .|| = ipix + ^ 2 x|| = pi+A 2 ||. 

So, the assertion is proved. 

Now we are in a position to show the following two propositions. 



□ 



Proposition 2 Let W be an arbitrary (n + 1) x (n + 1) orthogonal matrix with W{n + 
l,n + 1) = —a, < a < 1. Let W\\ = W(l : n, 1 : n) and \J3i, . . . ,/3 n , —a] be the last row 
of W . Then for S = diag([si, . . . , s n }) with s\, . . . ,s n arbitrary positive numbers ordered as 
< s~i < s 2 < • • • < s n , we have 



1 / <* + + m y/1 - a 2 - ft. 

C • ~ I ; —7T- ~T~ 



VT 



or 



< 



- c_:= + En - 



(19) 



Proof. Following (floT) . we get 

a _1 ^isi 

Ax 



Define the matrices 



U n + SlWl, . . . 



a 1 /3 n s r 

VT 



u n + s n w r , 



a 1 fiis\ _ a 1 (3 n s n _ 

rr^rt) ■ ■ ■ i , =■ i*r. 

yl — a 



'VT 



and 
Then 



A 2 = W u S - A x = [siwi, s n w n ] . 



Pi 



a 



VT 



From (I16p and (|17f) we obtain 



\w„ 



n-l 



\ i=i 



n 



VT 



and || [w\, . . . , w r , 



n-l 

£ 



Ujv: 



(20) 
(21) 



1. 
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Furthermore, since 1 1 jS' 1 1 = s n , it follows from 

\\s n Wn\\ < \\A 2 \\ < \\[wi, ... ,w n }\\ \\S\\ 

that 



Vi-« 2 -/^ 



VT 



s n < \\Ao\\ < s n . 



(22) 



or 



Note that A\A2 = O. Based on Lemma [T] and combining (|20|) . (|21|) with (|22p . we establish 
the desired inequality. □ 



Proposition 3 Suppose that < a < \, where a is defined as in Proposition^ Then ft 
c and c in Proposition O we have 



or 



c< c<4c. 



(23) 



Proof. If JMj < ^, it is easy to verify that 



v/i - « 2 -Jl > 1 



vT 



and that 



c > -c. 
4 



Thus. (23) holds. If > ^, then 



a 



so a 



-l IW 



=s n > s n , from which and the definitions of c and c it follows that 



c < 



< 



a-y ffi? + ... + ffi% 



1 \Pn\ _ 



2a- 1 J^ 1 2 s2 + ... + /3^ 



vT 



2a" l JP(si + ... + Pi~si 2 Jl -c?-Pl 

< " = h — ; — S n = 4C. 



vT 



a' 



Thus, d23J) still holds. □ 
This proposition means that the upper bound is at most four times of the lower bound 

accurately by its lower or upper bound 



in Proposition [2j So we can estimate 
within no more than four times of the exact 
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4 A closed formula for the TLS condition number 



Throughout the paper, we follow the definition of condition number in [5J [TB]. Let g : 
W — > R 9 be a continuous map in normed linear spaces defined on an open set D g C MP. 
For a given clq £ D g , clq ^ 0, with g(ao) ^ 0, if g is differentiable at ao, then the absolute 
condition number of g at ao is 

Kg (a ) = \\g'(a )\\, (24) 
and the relative condition number is 



<(a ) 



|g / (ao)lll|Qo| 
lb(ao)ll 



(25) 



where </(ao) G M qxp denotes the derivative of g at ao- 

In [Tj, a SVD-based closed formula for the condition number of the TLS problem was 
presented. Denote by n g (A, b) the absolute TLS condition number. It was shown in [1] that 



K g (A,b) = VWxTLsf + l D [V T , O n>1 ) V [D, O 



'n,l 



lT 



(26) 



where 



D 
D 



diag 
diag 



(7 



n+1 



(7 



n+1 



Next we will derive a new SVD-based formula for the TLS condition number. It is 
distinctive that, unlike (|26p that involves the singular values and right singular vectors of 
both [A, b] and A, our formula only uses those of [A, b}. 

Denote a = vec(A) and define the following function in a small neighborhood of 
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(>m(?i+l) 

a 
& 



XTLS = (A T A-a 2 n+l I n )- 1 A T b 



where a = a + vec(AA) = vec(^4), b = b + Ab. Then we have g([a T , b T ] T ) = xtls- Based 
on Theorem [H we can present the following result. 



Theorem 2 Given the TLS problem (pQ), let K g (A,b) and n g (A,b) be the absolute and 
relative condition numbers of the TLS problem, respectively. Then 



K g (A,b) = \\K\\, nf(A,b) 
where K is defined as in (|7|). 



\K\\\\\A, 



\XTLS\ 



(27) 



Proof. Recall that our TLS problem satisfies < o", t +i < a n . By Theorem Q] and the 
definition of g, we see that g is differentiable at [a T , b T ] T and g' {[a T , 6 T ] T ) = K. Then the 
assertion follows from (1241) and (1251) . □ 



Now we can establish our computable formula for the TLS condition number. 
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Theorem 3 Given the TLS problem ([T]), let [A, b] = J7diag(ai, . . . , a n+ \)V T be the thin 
SVD of [A, b] with V n = V(l : n,l : n). Then 



K g (A,b) = V\\xtls\\ 2 + 1 \\V n T S\\, 



(28) 



I % - — ^ 

where S = diagQsi, . . . , s n ]) with Si = z'_ i n+1 , i = 1, 



, n. 



Proof. Consider expression ([7]) of K. By the properties of Kronecker product of 
matrices, we get 



GG T = ([x^ LS , -1]® J m ) 
[In ® r T , O n>m ]G T = [I n ® r T , O n , m ] 



XTLS 
-1 



XTLS ® In 

I'm. 



= (lkTLs|| 2 + l)/ m , 
(I n ® r T )(x T LS ® Im) = XTLS r T 



and 



Thus, we have 



\In ® r j O n ru i 



I n ®r 

O m ,n 



{I n ®r T )(I n ®r) = \\rfl n . 



2AT ¥\\¥\\ G ~ aT ° ~ [In ® rT ' ° n,m] 



2G T r |P A-G T A 



I n ®r 

Om.m. 



= (WxtlsW 2 + l)A T A + ||r|| 2 / n - x TL sr T A - A T rx^ LS 
= {WxtlsW 2 + l)A T A + \\rfl n - 2o-1 +1 xtlsXtls, 

during which the last equality made use of the relation A T rx^ LS = o-^ +1 xtls x tls ^ na * ^ s 
obtained from ©. Denote P = A T A - a^ +1 I n . We get 



KK T = p- 1 ((\\xTLs\\ 2 + l)A T A + \\r\\ 2 I n -2a 2 n+1 x T LSX T ' LS )p- 1 



\XTLS\ 



+ l)^ 1 ( A T A + a 2 n+l I n - 2 f 1 ^ ) P-1 



XTLsf + 1) f P' 1 + f I n 



xtlsx tls 



p-i 



1 + IkrLsll 2 , 

during which the second equality used ((HJ). Denote V = [v\, . . . , an d note that 



(29) 
(30) 



n+l 



n+1 



[A, bf[A, b] - a 2 +1 I n+1 = ^2 afvivf - a 2 n+1 ^ 



Vivf 



i=l 



1=1 



i=l 
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We get 



i=l 



\ T 


In 




O 







[l n ,0][vi, . . . ,v n ] 
V u dmg(a 2 - a 2 n+1 , ...,a 2 n - a 2 n+1 )V^ := V U AV? V 





r t i 




v T 

n 



o 



(31) 



Similarly, since v n -\ 



^1+\\XTLS\\ 2 



XTLS 
-1 



In+1 



and 



1 



1 + || ZTLS II 2 



xtlsXtls ~ x tls 

-XTLS 1 



(c.f. (HOD), we have 



In+l ~ V n +lV^ +1 = [v 1 ,...,V n ] 



T X T LSX TLS _ T 

In ~ m 112 _ V H V 11- 

1 + FTL5 



(32) 



Combining (|31|) and (|32p . we have 



P- 1 ^?^ 2 p- 1 f T x tlsXtls \ p-l 

P +2<Tn+lP V n 1+\\XTLS\\ 2 ) P 

V^A-W,, 1 + 2a 2 n+1 (vjfA" 1 ^ 1 ) VuVl (v^A^V^ 
Vr/A-^ 1 + 2a 2 n+1 V u T A- 2 V 1 - 1 1 
V n T (A- 1 + 2a 2 +1 A~ 2 ) = (V U T S) (v n T S 



(33) 



□ 



Then it follows from (j3U[) and Theorem [2] that the desired equality holds. 

By Theorem [3l we can calculate K g (A,b) by inverting V^. Next we show that the 
condition number of is exactly yl + H^tlsII 2 - 



Theorem 4 Under the conditions of Theorem^ we have 
ai (V n ) = ■■■ =* n -i(Yn) = l,(r n (V n ) = 

and 



V^+JxTLsW 



11, 



ii> 



O-n(Vll) 



XTLS] 



(34) 



(35) 



Proof. By (fTDl) we get V(n + 1, n + 1) = — i _ 

yi+lFTisH 

, 1 < 1. Applying Proposition (TJ we prove the theorem. 

yi+\\ x TLs\r 



. Recalling that xtls 7^ 0, we get 

□ 
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5 Straightforward bounds for the TLS condition number 



5.1 Sharp lower and upper bounds based on the SVD of [A, b] 



In this subsection, we further investigate our result in Theorem [3] and show how to estimate 



n g (A,b) accurately without computing 



V^S , i.e., avoiding inverting V^. 
For the TLS problem ([1]), from now on we denote a = —V (n + l, n + l). By (jlOp we get 

1 



a 



\XTLS\ 



(36) 



Recalling that xtls 7^ 0, we have < a < 1. Then we have the following theorem. 
Theorem 5 For the TLS problem ([!]), it holds that 



k := a 



°i + cr, 



71+1 



<J7, 



< K g (A, b) < k := a 



-2 



o-i + o 



n+l 



'n+l 



err, 



a 



(37) 



n+l 



Proof. As before, let [A, b] = *7diag(<7i, . . . ,a n+l )V T be its thin SVD with V n = V(l : 
n, 1 : n). From (1341) it follows that 



o 



°n(V n T ) = MVn))' 1 = 1, V^T) = (o-niVn))- 1 

Define 5 = diagQsi, . . . , s n ]) with Sj = "^^ + ^" + -, i = 1, . . . , n. We then have 

CT i a n+l 

s n = <Tn(yn T )\\S\\ < \\V U T S\\ < \\V n T \\\\S\\ = a~ l s n . 



(38) 



□ 



Therefore, by Theorem [3] and (j36|) we get the desired inequality. 

We see that in (|37|) the ratio of the upper bound and the lower bound is — . As a 
consequence, both the bounds estimate K g (A,b) within — times. Therefore, if a G (0,1) 
is near to one, say ^ < a < 1, both the bounds are very tight and they estimate K g (A,b) 
accurately. We now present a family of examples to illustrate it. 



ep 


a 


0~n ~ °~n+l 


Kg(A,b) 




2 V U ™ +a n + l 


10" 3 


0.516 


9.89 x 1(T 4 


1.38 x 10 3 


1.37 x 10 3 


2.66 x 10 3 


10" r 


0.792 


9.99 x 10" 8 


8.93 x 10 5 


8.93 x 10 s 


1.13 x 10 y 


io- lu 


0.859 


1.00 x 10 _ 10 


8.24 x 10 9 


8.23 x 10 y 


9.59 x 10 9 



Example. We construct TLS problems as in [H Example 1]: Define 



[A, b]=Q 



S 

o 



yT € R mx(n+1) ) Q = j m _ 2yy T y = _ 2 ^T 



where y G M m and z G R n+1 are random unit vectors, and £ = diag([n, n— 1, . . . , 1, 1 — eJ) 
for a given parameter e p . Fixing m = 100, n = 20, and taking e p = 10 -3 , 10 -7 , 10 -10 , 
respectively, we get three different TLS problems. As indicated by the results of K g (A,b), 
as e p decreases, the TLS problem becomes more ill conditioned. This is also reflected by 
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the decay of a n — cr n +i; see [6] and Theorems [BHSl Since a's are near to one, the lower and 
upper bounds in (f37|) estimate the TLS condition numbers accurately. 

In view of (|37p and the followed comments as well as the above example, it is only 
possible and significant to improve the bounds essentially for the case that a is small 
relative to one. Without loss of generality, we assume that 

< a < ~. (39) 

It will appear that we can establish some lower bound k and upper bound R such that 
k < R < Ak holds. As a result, together with (|37p . we can estimate the TLS condition 
number K g (A,b) accurately. 



Theorem 6 Given the TLS problem ([I]), let [A, b] = L r diag(o'i, . . . , a n +i)V T be its thin 
SVD. Denote [ft,.. 
i = 1, . . . , n. Then 



SVD. Denote [/Si, ... , /3 n , —a] be the last row ofV and S = diagQsi, . . . , s n ]), Sj = " + ~ 



2 I VT 



O: 



2 



yl — a 

Moreover, ifO<a<^, then 

K < R < 4k. 



Proof. Recall that < a < 1. Noticing that < s\ < S2 < • 
Proposition [21 we have 



< s n and applying 



+ 



< 



V U T S 



or 



where Pil = V(l : w, 1 : n). By Theorem (3[ we get the first part of the theorem. Further- 
more, we obtain the second part of the theorem by Proposition [3l □ 

The significance of this theorem is that we can estimate the condition number n g (A, b) 
accurately and cheaply by its lower or upper bound without expensively inverting in 
order to compute ||V]J T if?||. 



5.2 Lower and upper bounds based on a few singular values of A and 

[A, b] 

In [15], some bounds for the condition number of the Tikhonov regularization solution have 
been established using only a few singular values of A, where A is the coefficient matrix 
of the least squares problem under consideration. Such kind of bounds are particularly 
appealing to the TLS problem since, for a large scale TLS problem, its condition number 
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formula in Theorem [3] and the bounds in Theorem [6] involve all the singular values of [A, b] 
and are impractical for computational purpose. 

Actually, we have presented such kind of bound ([3T|) . but as we commented there, for 
small a, the bounds may overestimate or underestimate K g (A,b). In the following, we 
establish some new lower and upper bounds in the same spirit and finally achieve sharper 
lower and upper bounds for the case of < a < \. 

Theorem 7 For the TLS problem (pQ), we have 

Hi < K g (A,b) < Ri, (40) 

where 



- 1 ~ P ' K l - & 2 -2 • ( 41 ) 

°n-l °n+l °n °n+l 

Proof. Recall that r = Axtls — b. Denote 

M = (A T A - o- 2 n+l I n )- 1 {(WxtlsW 2 + 1)A T A + \\rfl n ) (A T A - a 2 n+l I n )-\ 
From (|29p . we have 

KK T = M — 2a 2 n+l (A T A - a 2 n+l I n )~ l x TLS xl LS (A T A - a 2 n+1 I n )-\ (42) 

From now on, denote by Aj(M) the ith algebraically largest eigenvalue of M, where M is 
an arbitrary symmetric matrix. By the Courant-Fischer theorem [9j p. 182], we get 

A 2 (M) < Ai (KK T ). (43) 

Furthermore, since 2a 2 +1 (A T A — a 2 +1 I n )^ 1 XTLSx'^ LS (A T A — o- 2 +1 / n ) _1 is nonnegative 
definite, the following inequality holds 

\i{KK T ) < Ai(M). (44) 

Combining (|43p with (|44p and based on (|27p . we have 

v/A 2 (M) < k 9 (A, b) < v/Ai(M). 

It is easy to verify that the eigenvalues of M form the set 

r(||x TLS || 2 + l)a] + ||r|| 2 \ n 



We define the function 



, . , vl krL5|| 2 + iy 2 + ||r|| 2 
Ha) = _ , a , .,, + ., 

and differentiate it to get 



.,, , -2a 3 (\\xTL S f + 1) - MWxtlsW 1 + 1K 2 , +1 - MV\ 
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It is seen that h'(a) < and h(a) is decreasing in the interval (cr n +i, oo). Thus, we get 
ni x TLS \\ 2 + l)a 2 n + \\r\\ 2 x , . (||x TL5 || 2 + l)a2_ 1 + ||r|| 2 



Ai(M) = ^^A_^J±_^ } A 2 ( M ) 



and 

kTL5|| 2 + 1)<^_1 + ||r|| 2 v«FTl)ef 



< « ff (A,6) < 



a 2 -a 2 -™g\™,"j^ cT 2 -rr 2 

°n-l °n+l °n <J n+l 



Recalling that 1+ ||j.y L9 ||a = fn+i> we complete the proof. □ 
Remark. In Corollary 1 of [T], the authors prove that 



\A + \\XTLS\\ 2 J(T 2 + (?n+l 



Kg(A,b) < o 



Since a n < a n and a n < ai, we get 



V 1 + \\XTLS\\ 2 JV 2 + <?l +l 

«1 < — 2 _ 2 • 

°n a n+l 

Therefore, our R% in (|4ip is sharper. Moreover, the improvement must be significant when 
f L > 1 - 

u n 

It is seen that the lower and upper bounds in Theorem [7] are marginally different pro- 
vided that <r n and <7 n _i are close. This means that in this case both bounds are very tight. 
For the case that a n and a n —l are n °t close, we next give a new lower bound that can be 
better than that in Theorem [3 

Theorem 8 It holds that 

k 2 < K g (A,b) < K%, 
where R\ is defined as in Theorem [?] and 



_ a/1 + H^TLsll 2 



Moreover, when <r n _i > cr n+ i + y & 2 — <J^ +1 , we have 
Proof. Denote P = A T A - cr 2 +1 I n . From (j3QJ), we have 



ijj— KK T = P- 1 + 2a 2 n+lP -i (l n - .Tf 4 ™ ) P- 1 - 



Since the second term in the right-hand side of the above relation is positive definite, we 
have 

(\\x TLS \\ 2 + l)Xi(P- 1 ) < Xi( KK T ), 
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that is, 



XTLS\? + 1 < K 2 (A,b), 



which used (j27f) . Thus, we obtain the first part of the theorem. 
The second part of the theorem is obtained by noting 



r n-l + a n+l 1 1 

< < 



rr '"- I - ,T "- ! - < T <- 1 , iai - cr 2 n+1 



under the assumption that <r n _i — <J n +i > y &\ — a n+v n 

Remark 1. At first glance, the assumption in the second part of the theorem seems not 
so direct but we can justify that it indeed implies that a n and o n —\ are not close. Actually, 
it is direct to verify that the second part of Theorem [8] holds under a slightly stronger but 
much simpler condition that 

Remark 2. From 



1 + 



7 n + l 



T 2 



it is seen that ^ > 1 provided <r„ + i > 0. Only for a n+ i = 0, k± = k 2 holds. At this time, 
b G Tl(^) and r = 0. 

We observe that the bounds for k 5 (^4, 6) in Theorem[8]are tight when is considerably 
smaller than one. On the other hand, if g +1 is not small, these bounds may not be tight. 
For this case, we will present a new upper bound that may better estimate K g (A, b). 

Keep (f36|) and ([39]) in mind. Based on Propositions [2H3J we establish the following 
theorem. 



Theorem 9 IfO < a < \, then 



\A + \\XTLS\\ 2 f A U\ / - / l + 31p 2 yT+ . v 

k 2 := < K g (A,6) < w 2 := */-: s / 45 



where p = . 



Proof. The lower bound is the same as that in Theorem [8] We only need to prove the 
right-hand side of ([35]). As before, let [A, b] = f/diag(<7i, . . . , a n+ i)V T be its thin SVD with 
Vu = V(l : n, 1 : n). From JSTJ and ([35]), we get 

' KK T = P- 1 + 2a 2 P- 1 ( I n - f T T^ ) P-\ 



\XTLS\\ 2 + 1 n+1 * V" 1 + \\XTLS\\ 2 

+ 2 C r2 +1 F n T A-V n 1 := P' 1 + C, (46) 
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where P = A T A - a 2 +1 I n , A = diag([crf - a 2 +1 , . . . , o£ - a^ +1 ]). Denote 

Cn+l 



D = diag([di,. . . ,d n ]), di 



diag([ii, . . . ,t n ]), ti 



tr% - a, 
1 



2 ' 
n+l 



i= l,...,n, 



,i = 1, . . . ,n. 



o? — a. 



n+i 



Then P- 1 = fV5J T rl (T^ 1 ) and C = 2 (V^-DJ (W^ 1 ). 

Note that < d\ < di < ■ ■ ■ < d n and < t\ < t2 < ■ ■ ■ < t n . From Proposition [2J we 

get 



and 



\/T- a 



< 



V U T D 



< 



2 Vi^? 
a -^p{dl + ... + pidl 



+ 



En 



< 



, a- V^tjf + . . . + (ft* 



(47) 



respectively, where . . . , /3 n , —a] is the last row of V cis defined previously. Define Av, 
£t = f5±j= . Then 



*1 



Thus, from (|4"T|) we get 



Cn+l 



<C fen 



d n -i 

tn-l 



Cn+1 



2 2 
°"ti-1 ~~ °n+l 



5: &71- 



71 I|C|1 ' 



a: 



Vffftf + ... + /%% 



(48) 



For the lower and upper bounds on 



V U T T 



above, Propositions [2H3] tell us that 



+ t n < 2 
< 4 



+ 



y n T r 



(49) 



Therefore, based on (1481) and (1491). we obtain 



-±=||C||5<4Av, 



4A;„,||P" 1 ||5 



and 



2|| D-l| 



l|C|| < 32#||P 



(50) 
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Combining (|46p with (|50|) and based on (|27j) . we get 

k p (A,6) = H^ll = ||iOT T ||t < ^l + 32A;2Vl + ||xrLs|| 2 ||^ 1 ||^ 



l + 31p 2 ^1 + ||x TLS || 2 

i - p 2 n~2 ~2 

' ' °Vi °n+l 



□ 

Remark. It is clear that the bounds in Theorem [9] are tight when p = CT " +1 is con- 
siderably smaller than one. Note that the lower and upper bounds in Theorem [8] differ 
considerably when <J g +1 is close to one. The result in this theorem is of particular impor- 
tance when is close to one, since the upper bound here can be considerably sharper 
than that in Theorem [8] if only p = is small. 

The improvement of R<i over R\ can be illustrated as follows. For p = " +1 small, we 
have 



^ lUA „„_ _ /l + 31/> 2 ^/1 + WxtlsW 2 \\[A,b]\\ F 



■\\[A,b]h 



\xtls\\" 1 ~' " J " J V !-P 2 Ikrisll . /#2 _ a 2 ' 

u n u n+1 



an upper bound for K rel (A, b), is a moderate multiple of , 1 2 , while 



^TiS RTiS °"n - °" 



n+1 



is a moderate multiple of , 2 1 z — . So the improvement of Kn el over «^ el becomes significant 

°"n CT n + l 

as cr ra+ i and a n are close. 

Golub and Van Loan [6] derive an upper bound for the relative condition number for 
the TLS problem, which, in our notation and case, is simplified as 

rel / a i\ 9(71 \\b\\ \ \\[A,b]\\ F 

Ktls(A b) := 1 + . (51) 

&n — 0n+l V ^n-^n+l/ ||0|| — CT n +l 

We will numerically illustrate improvements of our bounds K^ el and R™ 1 over (|5ip in the 
next section. 



6 Numerical experiments 

We present numerical experiments to illustrate the tightness of the bounds in Theorems 
EHS] and to show that our upper bounds can be much better than Golub- Van Loan's upper 
bound ([5T]) in [6j. For a given TLS problem, the TLS solution is computed by @. All the 
experiments were run using Matlab 7.8.0 with the machine precision e mac h = 2.22 x 10~ 16 
under the Microsoft Windows XP operating system. When a is near to one, it is trivial 
to estimate the TLS condition number since the lower and upper bounds in (|37p are both 
sharp in this case. Next we will be concerned with only the case that a is not near to one. 
For all the test problems, we always have < a < 5. 
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Example 1. The data A £ jR"»x(m-2) } & e R m are taken from jjgj. 



So the exact 



TTl — 1 


_1 


-1 




-1 


-1 


m — 1 • • • 


-1 




-1 


-1 


_1 


m — 1 


, 6 = 


-1 


-1 


_1 


-1 




m — 1 


-1 


_1 


-1 




-1 


"1,-1, 


..,-l] T €l 




= y/2m 





m 








rcl 




Rf l 


K TLs(A, b) 


200 


7.07 x 10~ 2 


7.07 X lO" 1 


2.01 x 10 2 


2.00 x 10 2 


2.15 x 10 2 


3.46 x 10 2 


5.15 x 10 3 


500 


4.47 x 10~ 2 


7.07 x lO" 1 


5.01 x 10 2 


5.00 x 10 2 


5.15 x 10 2 


8.65 x 10 2 


1.21 x 10 4 


1000 


3.16 x 10~ 2 


7.07 x 10- 1 


1.00 x 10 3 


1.00 x 10 3 


1.02 x 10 3 


1.73 x 10 3 


2.35 x 10 4 



Table 1: Example 1 



In Table [H we list the results of the relative TLS condition number Kg (A, b) and its 
bounds 



.rel 



D.2 



\XTLS\ 



\A, b] 



-.rel 



K 2 



\XTLS\ 



■II [A, 



«1 



\XTLS\ 



■II [A b]\\i 



and KjT LS (A,b) (see (f5T|) ). where k 2 an d K 2 are defined in (|43|) . ki is defined in (|4"T|) . 

We can see that the test TLS problems are well conditioned. Both the distance of <r n +i 
and o" n and that of a n +i and a n are not very small so that k^, R r 2 el and R™ 1 all estimate 
K r g el (A,b) accurately. Kjr LS (A, b) is also good but is not as good as our bounds, and it 
overestimates K r g el (A, b) by one to two orders. 

Example 2. In this example, we take the TLS problem from [12J. Specifically, an 
m x (m — 2oj) convolution matrix T is constructed to have the first column 



V2^exp| - ( "-' +1) ' 



and the first row 



t 







i = 1,2, . . . ,2u + 1, 
otherwise, 

if J = l, 
otherwise, 



where /3 = 1.25 and w = 8. A Toeplitz matrix A and a right-hand side vector 6 are 
constructed as A = T + E and b = g + e, where g = [1, . . . , 1] T , E 1 is a random Toeplitz 
matrix with the same structure as T and e is a random vector. The entries in E and e 
are generated randomly from a normal distribution with mean zero and variance one, and 
scaled so that 

||e|| = 7||<?||, ||£|| = 7 ||f ||, 7 = 0.001. 
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m 


0"n+l/0"n 




K f(A,b) 


K rel 








100 


0.964 


1 - 6.08 X 10~ 8 


3.41 x 10 7 


3.24 X 10 7 


6.68 x 10 8 


1.31 x 10 11 


4.70 x 10 17 


300 


0.996 


1-3.67 x 10~ 8 


8.11 x 10 7 


7.21 x 10 7 


4.45 x 10 9 


3.76 x 10 n 


2.03 x 10 i9 


500 


0.998 


1 - 6.76 x 10~ 8 


8.01 x 10 7 


7.07 x 10 7 


6.36 x 10 9 


2.72 x 10 n 


4.13 x 10 i9 



Table 2: Example 2 



In this example, for each test TLS problem, we compute the same quantities as those in 
Table [TJ As indicated by the k t ^(A, 6)'s, these TLS problems are all ill conditioned. Their 
ill conditioning is also reflected by the fact that (J n +i and cr n are close. As estimates of the 
relative condition number K r g e ^(A,b), both the lower bound k^ 1 and the upper bound R r 2 el 
are sharp since <J n +\ and a n are not so close, but the upper bound R\ el is not tight any 
longer and overestimates K r g el (A, b) by about four orders. Even though it is not satisfying, 
/^ el is still much better than K^ LS (A,b), which severely overestimates n g el (A,b) by ten to 
twelve orders. We also comment that R r ^ improves «^ el by two orders. 



* 
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O 
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+ 

6 


6 


□o 












+ 
o 


-7 


-6 


-5 


-4 


-3 


-2 



log"! 0(a) 



Figure 1: log w (^{A,b)) (+), logwinf) (°), log 10 (Rf l ) (o), log 10 (K? 1 ) (□), and 
log 10 (K^ LS (A,b)) (*) for (m,n) = (500,350). 

Example 3. Keep in mind that the distance between cr n +i and a n can control the 
conditioning of a TLS problem; see Theorems [HHSl or [6]. In this example, we compare 
the bounds «2 , «2 , ^i el and K r ^ LS {A,b) for various distances between a n+ ± and a n . For 
given m, n, we construct A and b by taking different values of a. Specifically, we generate 
randomly (with standard normal distribution) two n x n orthogonal matrices U, V, and 
one m x (n + 1) matrix C\, and then compute C\ = WSV^, the thin SVD of C\. Having 
these fixed matrices U, V, U and E, we take a = 10 -2 , 10 -3 , . . . , 10 -7 , run the function 

C = generator(n, a, U, V, U, E), 

and set A = C(:,l : n), b = C(:,n + 1), respectively. See the pseudo-code function in 
Appendix for details. In such a way, with (m, n) = (500,350) and (1000, 750), we generate 
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100 samples for each a, respectively. For each set of TLS problems with the same a, we 
compute K r g el (A,b), k^ 1 , k™ 1 , ^ el and K T rp LS (A : b). We plot the (log scale) averages of these 
quantities and the corresponding (log scale) a in Figures HH21 We also report the averages of 
1 — , a measure of the distance between <7 n+1 and a n , in Table [3l We comment that the 
averages of ^±1 for (m,n) = (500,350) and (1000,750) are 0.942 and 0.959, respectively. 




-5 -4 -3 

log"! 0(a) 



Figure 2: log 10 (K g cl (A,b)) (+), log lQ {^) (o), log 10 (Rf) (o), /o<?ioK cl ) (□), and 
log 10 (K r ^ LS (A,b)) (*) for (m,n) = (1000,750). 



(m,n) = (500,350) 



a 


10~ 2 


10~ a 


10" 4 


10" b 


10~ b 


io-' 


1 0"n + l 


3.00 x 10" 4 


3.01 x 10 _ti 


3.01 x 10" 8 


3.01 x 10~ 1U 


3.01 x 10" 12 


2.94 x 10" 14 


(m,n) = (1000,750) 


a 


10~ 2 


10~ a 


10" 4 


10~ 5 


io- b 


IO"' 


^ 0"n + l 


3.62 x 10~ 4 


3.63 x 10~ B 


3.63 x 10~ s 


3.63 x 10~ 4U 


3.62 x 10~ 12 


3.54 x 10~ i4 



Table 3: Example 3 



We can see from Figures HH2] and Table [3] that as a decreases, a n and a n+ \ become 
closer, and the TLS problem becomes more ill conditioned. k^ ls (A, b) always severely 
overestimates K g el (A, b). For a = 10~ 2 in which a n and cr n+ i are not very close, R^ 1 is tight 
and estimate K v g l (A,b) accurately. For a = 10 -3 , & n and cr n+1 are closer. In this case, 
is no longer tight and estimate n T g l (A, b) poorly but it still improves K^ LS (A,b) by 
four orders. We observe from Figures EH that Rf l and K TLs(A,b) estimate K r g el (A, b) more 
poorly as a decreases. Even so, R\ el is always smaller than k^ 1 ls (A, b) by about four orders. 
In contrast, for all the cases, since a n and <J n +\ are not so close, k?' and S 61 estimate 



,.rcl 



(A, b) accurately. Particularly, for a = 10 



the upper bound n 2 



rcl 



improves both k\ 



rcl 



and K^p\ jS (A,b) very considerably by many orders. 

Example 4. In this example, we generate the entries of A and b as random vari- 
ables normally distributed with mean zero and variance one and observe k^ 1 , ^2 



;rcl 



and 
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(m, n) 


Kf(A,b) 


1 

„rei 
!±2 


1 





K TLs(A, b) 


(200, 75) 


1.33 x 10 a 


4.85 x 10 2 


1.07 x 10 4 


1.35 x 10 5 


8.45 x 10 8 


(500, 350) 


1.53 x 10 4 


6.86 x 10 3 


1.09 x 10 5 


2.08 x 10 Y 


3.40 x 10 1U 


(1000,75) 


3.34 x 10 3 


7.28 x 10 2 


2.20 x 10 4 


4.41 x 10 5 


2.34 x 10 9 


(1000,750) 


3.32 x 10 4 


1.26 x 10 4 


2.60 x 10 5 


1.14 x 10 Y 


5.47 x 10 1U 



Table 4: Example 4 



k tls(A, b). For each (to, n), we conducted 100 random experiments. We report the average 
results of 100 experiments in Table [H We observe that, as estimates of k™ 1 (A, b), both k^ 1 
and «2 el are tight. The upper bound k?> c1 improves K^ el by one or two orders and improves 
K r ^ l LS (A,b) by about five orders. Clearly, the test TLS problems are quite well conditioned, 
but K^p l LS (A, b) is a rather poor upper bound and overestimate K r g el (A, b) too much. 



7 Concluding Remarks 

In the paper, we have studied the SVD-based condition number theory of the TLS problem. 
For the TLS condition number, we have established a new closed formula. Starting with it, 
we have derived sharp lower and upper bounds. Importantly and more practically, we have 
presented both lower and upper bounds that use only the smallest two singular values of A 
and of [A, b]. We have demonstrated the tightness of our bounds by numerical experiments. 
Throughout the paper, the considered TLS problem is assumed to satisfy condition ([2]) and 
has a unique TLS solution. It is significant and important to extend the results presented 
in the paper to a general generic TLS problem |19} [20] that has non-unique TLS solutions 
or to the non-generic TLS problem [19]. We will consider these problems in forthcoming 
papers. Besides, it might be worthwhile to investigate how to apply the core problem theory 
|17j to study the TLS condition number. 



A A pseudo-code of generating test TLS problems 

For the TLS problem ([I]), Lemma 4.3 in i6] gives 

r < \\XTLS\\ < ~ • 

Z[a n — <r n +i ) cr n —a n +i 

Equivalently, it holds that 

\ulb\ \\b\\ 

-rr, [7 - °™ ~ a ri+l < 71 [7- (52) 

2\\XTLS\\ \\ X TLS\\ 

Denote V(n + l,n + 1) = —a and recall that a = , 1 From (1521) we see that a 

\/i+\\ x tls\\ 2 

small a implies that a n and cr n+ i are close in some sense. 

The following pseudo-code generates C = [A, b], where A £ ]j mxn anc | 5 g jj"^ ft 
the thin SVD C = [A, b] = UT,V T that satisfies V(n + 1, n + 1) = -a, where a is a given 
parameter such that < a < 1. The idea of construction comes from Proposition [H 
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C = generator(n, a, U, V, U, S) 

% n : a given positive integer 

% a : a given positive number with < a < 1 

Vo U, V : two n x n orthogonal matrices 

%P:anmx(ii + l) column orthonormal matrix 

% £ : an (n + 1) x (n + 1) diagonal matrix with positive diagonal elements 
% Output : C= [A 6], where A = C(:,l : n),6 = C(:,n + 1). 
u = C/(:,n); 
trf = V-(:,n)'; 

Vii = £/(:, 1 : n - 1) * V(:, 1 : n - 1)' + a * u * vt; 

t = sqrt(l — a * a); 

V=[ [V u ,t*u\;[t*vt,-a] ]; 
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